WORLD INTELLECTUAL PROPERTY OROANIZATION 
International Btueau 



PCT 

INTERNATIONAL APPLICATION PUBUSHED UNDER THE PATENT COOPmiATION TREATY (PCT) 


(51) International Patent Classification ^ : 
H04N 7A8, GOCT 1/60, H04N 3/15 


Al 


(11) International Publicatloa Number: WO 98A)1999 

(43) International Publication Date: 15 Januaiy 1998 (15.01.98) 


(21) International AppUcaUcm Number: PCr/NO97/00175 

(22) International Filing Date: 9 July 1997 (09.07.97) 


(30) Priority Data: 
9614434.0 


10 July 1996(10.07.96) 


GB 


(71) Applicant (for all designated States except US): SECOND 

OPINION SOLUTIONS AS [NO/NO]; Sagvn. 21E, N-0458 
Oslo (NO). 

(72) Inventor; and 

(75) Inventor/Applicant (for US only); CLINCH, Noah [GB/GB]; 
Fairfield Imageing Ltd., 8 Lonsdale Gaixlens, TUmbridge 
WcIU, Kent TNI INU (OB). 

(74) Agent: ONSAGERS PATENTKONTOR AS; P.O. Box 265 
Scntnim. N^103 Olso (NO). 


(81) Designated States: AL, AM. AT, AU. AZ» BA, BB. BG. BR. 
BY, CA. CH, CN. CU. CZ. EHE. DK, EE. ES, FI, GB. GE, 
GH, HU. IL. IS. JP. KE. KG. KP, KR, KZ, LC. LK, LR. 
LS. LT, LU, LV. MD. MG. MK. MN, MW. MX. NO. NZ. 
PL. PT, RO, RU. SD. SE, SG, SI. SK. SU TJ, TM. TO. 
TT. UA. UG. US, UZ, VN, YU, ZW. ARIPO patent (OH, 
KE, LS. MW. SD, SZ. UG. ZW). Eurasian patent (AM. AZ, 
BY, KG. KZ. MD, RU. TJ, TM), European patent (AT. BE, 
CH, DE, DK. ES, FI, FR, GB, GR. IE, IT. LU. MC. NL. 
PT, SE), OAPI patent (BF, BJ, CF. CG. CI. CM, OA, ON. 
ML, MR, NE, SN, TD, TO). 


Published 

With intemathnai search report. 

Before the expiration of the time limit for amending the 
claims and to be republished in the event of the receipt of 
amendments. 


(54)1106: VIDEO DISPLAY SYSTEMS 


14- 




-15 


-18 


-19 


(57) Abstract 

A method for transmission 
of a sequence of high quality im- 
ages for display on a visual dis- 
play unit is described. It has par- 
ticular application in the field of 
telepa^ology. where magnified 
images obtained by scanning a 
medical specimen on the stage of 
an optical microscope arc trans* 
mitted from a local pathologist 
to a remote consultant for diag- 
nosis. The method consists of 
the steps of advancing the field 
of view of the camera in dis- 
crete steps across the object be- 
ing viewed to capture a sequence 
of contiguous images, temporar- 
ily storing a digital representa- 
tion of a first of the inuiges in 
a first buffer store and a digi- 
tal representation of at least pait 
of the next succeeding image in 
a second buffer store, control- 
ling the data In the two buffer 
stores such that the data follows 
the leading edge of the advanc- 
ing viewport and data in the first 
store representing a trailing in- 
cremental strip of the first stored 

image is progressively discarded while data in the second store representing a trailing incremental strip of the next contiguous image is 
transferred to a location in the first store occupied by the data representing leading incremental strip of the first stored image, and reading 
out the contents of the first buffer stme to the display unit. 
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VTORQ DISPLAY SYSTEMS 

This invention relates to the transmission of a sequence of high quality images for 
display on a visual display unit. U has particular application in the field of 
5 tclepathology where magnified images obtained by scanning a medical specimen on 
the stage of an optical microscope are transmitted from a local pathologist to a remote 
consultant for diagnosis. 

For general diagnostic practice in tumour histopathology and cytopathology, because 
10 of the potentially serious consequcxxcs of misdiagnosis, it is hardly ever acceptable 
to examine just one or a few static images from the microscope, no matter how high 
their individual quality may be. Instead, it is accepted that the diagnostician must be 
free to examine any pan of the specimen, at any of the magnification factors which 
the microscope allows. Thus the remote consultant should have the ability to 'scan* 
IS the field of view across the specimen on the microscope stage; both along the left- 
right (-x) axis and the top-bottcnn (-y) axis of the field of view. The consultant 
should also be able to move the stage in the longimdinal (-z) axis of the microscope 
to adjust the focus. 

20 Controlling the microscope stage throu^ verbal instructions to a local pathologia is 
unacccptably slow and unreliable. Transmitting a complete set of images which 
together cover the whole ^)ecimen could be done automatically using a motorised 
stage and aiitable camera control software, but would require die transmission of 
arcHind 4000 images for a histopathology section of ISmm by 10mm. This again is 

25 unacceptable. 

Accordingly, remote contrd of die sender's microscope is virtually essential far a 
practical system. 

30 One such system is described, for example, in US-A-5,216,596 and US-A-5,297,034. 
In this known system, the magnified image of the specimen is recorded by a video 
camera and converted to an electronic video signal which is then trausiniued over a 
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communication link to a remoce video display monitor. Control signals are generated 
by a computer processing unit at the remote workstation for remotely controlling the 
functions of the microscope, including motorised stage movement, magnification, 
focus and illumination control. 

5 

The main problem with diis known system is that the quality of die image viewed by 
the rem(xe consultant on the display monitor is well below the quality that would be 
seen by viewing the specimen direcdy through the microscope. 

10 There is an emerging ccmsensus that 1024 x 768 is the minimum acceptable pixel 
format for di^lay of diagnosdc-quality images on a high-resolution colour monitor. 
This rules out the use of all analog video cameras, monitors and image 
compicssiofi/decomprtssicMi devices (codecs) which art based on broadcast standards 
such as PAL or NTSC. The bandwidth limitaticm imposed by the broadcast standards 

15 reduces the effective pixel number in each image to about a quarter of the number 
required fcH* diagnostic resoiuticm; in additicm, in composite TV equipment, die colour 
resolution of the signal is further redwed by ±e chrominance subsampiing. 

Although digital videocameras with CCD chips are now available which are suitable 
20 for capturing high-resolution microscope images, these cannot be used to directiy 
display images on composite video monitors. Instead they are desgned to work widi 
image digitisers (frame grabbers) by means of which a digital representation of the 
image is stored in RAM, or on storage medium such as magnetic (Use or CD ROM. 
To visually di^lay such an image, it must be written to an *RGB* colour monitw 
25 with a display driver capable of handling images of at least 1024 x 768 pixels at 8 or 
more bits/colour channel. 

The overriding problem in the use of digital cameras in a telepathology diagnostic 
system is that, unlike analog video cameras, digital cameras cannot provide images 
30 ai video rate: in fact a maximum rgb frame rate of about 1/2S of video rate (2/sec), 
is typical, and in several cases the frame rate is less than 1/sec. 
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Although standard analog TV videocodec technology can be used to compress the 
video images such that the required bandwidth is reduced to a practical level (say 384 
kbit/sec), if 50 images must be transmitted per second, this allows 384/50 or about 
8 kbits per image. Since the final Image Is normally reduced to a size of (512 x 384) 
5 24 bit pixels (or about 4 Mbit), this recpiires ihe codec to perform Oossy) image 
compression of 500: 1 . This can only be achieved at the expense of significant image 
degradation. 


To achieve smooth scicWing of the field of view of a digital videocamera across a 
10 specimen, the image refresh rate in the viewport must be comparable to At flicker 
fiisioa ftequency of normal human vision, say 30/sec. But because the maximum 
image capwrt rate of the digital camera is much less than this (of the order of 
Iftame/aec), it is not possible to simply grab, transmit, and display a stream of 
complete high-resolution images as the specimen is scanned under the objective. To 
15 overcome this ftindamenial (flfficulty without loss of image resolution, the present 
invention makes use of the fact that during scrolling, between one screen refresh and 
the next, although the rgb values of every pixel will in general be changed, die bulk 
of the display is simply shifted slighUy in a vertical or horizontal direction so that the 
information content of tf»e image as a whole remains almost constant. 

20 

According to the present invention tiiere is provided a method of diq)laying a 
sequence of images from a digital videocamera on a video display unit, the method 
comprising advancing die field of view of the camera in discrete steps across die 
object being viewed to capnire a sequence of contiguous images, temporarily storing 

25 a digital representation of a first of tiie images in a first buffer store and a digital 
tq>iesentation of at least part of die next succeeding image in a second buffer store, 
controlling the data in die two buffer stores such diat die data follows the leading edge 
of die advancing viewpon and daa in die first store representing a trailing incremental 
strip of Uie first stored image is progressively discarded while data in die second store 

30 lepreaenting a trailing incremental strip of die next contiguous image is transfened to 
a location in die first store occupied by die data representing a leading incremental 
strip of die first stored image, and reading out die contents of die first buffer store to 
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the display unit. 

The second buffer may either hold data representing a complete new image in the 
camera viewpon, or data representing an increment or * scroll quanta' of the image. 
S Where the second buffer holds an entire image, an intermediate or 'scroll' buffer 
containing the 'scroll quanta' may be inserted between the first and second buffers. 
Where the second store contains an entire image, the incrementing is continued until 
die entire contents of the first buffer store representing the original image stored 
therein have been discarded, and a new image is then grabbed imo the second buffer 
10 store and the process is repeated. Alternatively, the new im^ge is stored in a third 
buffer store and the contents of the second buffer store are then replenished 
continuously with increments frcmi the third buffer store. In either case* the 
specimen appears to be scrolling smoothly across the field of view while in reality it 
is moved in a series of quick steps followed by culture of contiguous images. 

15 

The data entered into the first buffer r^resenting the trailing incremental strip of the 
image stored in the second buffer stcnre can be entered direcUy from the second buffer 
store or throu^ an intermediate scroll buffer containing only the incremental portion 
of the image being transferred. 

20 

When scrcrfling in the y*direction, the progressive r^lenisbment of the image in the 
first store is performed line by line or in multiples of one complete line, whereas 
during x-scrolling each line of the di^Iayed image is created as a composite of the 
corresponding lines in the image buffers. 

25 

For a colour rgb image, first and second buffers can be provided for each of the three 
colour components, and the readout is to a single common display buffer. 

By way of example only, an embodiment of the invention will now be described widi 
30 reference to the accompanying drawings in which: 

Fig. 1 is a schematic block diagram of a remote control celepathoiogy system; and 
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Fig. 2 is a diagram illustrating the y-scrolling of the image in the system of Fig. I. 

Referring to these figures, the pathological specimen to be viewed Is placed on a 
microscope stage 10 beneath an objective tens 11 of the microscope 17 and 
5 illuminated by a light source 1 6. A digital video camera 12 with a CCD chip aiitable 
for capturing high-resolution microscope images is fitted to die microscope. 

A ditital interfece card (frame grabber) 13 retrieves and stores a digital represeniation 
of the image vievved by the camera 12, and Ais digital representation is processed by 
10 a local computer 14 and transmitted to a remote computer 15 operated by a 
consultam. 

The remote computer 15 is linked through a display driver 18 to a video display unit 
19. 

15 The microscope sage 10 is remotely controlled by the consultam through a stepper 
motor unit 20. A similar situation holds in the case where the consultant wishes to 
use the computer 15 to control a local microscope (not shown). As the consultant 
drives the microscope stage in discrete steps under die (fixed) objective 11 of the 
microscope 1 7, it appears as though a viewport of fixed size is being moved (scrolled) 

20 smoothly across a single very large image of die specimen (referred to herein as a 
'virtual digital image'). 

The manner in which apparent scroUing of the viewpon across die virtual digital 
image is achieved is Illustrated in Fig. 2. Scrolling can take place in bodi directions 

25 along trajectories parallel to eidier d» 'y'-axis or to the 'x'-axis. For simplicity 
we will first describe die method used for downward scrolling across a 'virtual digital 
image' made of uncompressed monochrome or interieaved-cdour images. This 
mediod can be easily extended to scrolling in any direction across any type of digital 
image, including compressed and noo-interleaved colour images, and including Images 

30 with any number of bits per pixel. 

In the following description, the number of Wts^pixel in each uncompressed 
image is called 'ribits' . The number of horizontal 'lines' in each image will 
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be denoted by *ymax\ and the number of pixels within each of these 
horizontal 'lines* by *xniax\ Thus xmax, ymax are the digital width and 
height of the images respectively and the bu^er size required to hold a single 
image is ^ven by the product (xmax. ymax. nbits). 

3 

Fig. 2 achemadcally rq>resents two contiguous digital images, continuous in the 
specimen's 'y*-directiOT which have been acquired into buffer stores buf,0 and bt^^ 
L The data buffers are continuous blocks of RAM, allocated at stan-up by the 
operating system of the computer at the request of the controlling program. The 
10 individual (x) scan*lines in the images are denoted a, and A, B,C*.. in bitfj) 
and bufj respectively. Scanning commences at a pcknx in time when the Image in 
bitfJD is already being displayed. 

SLrmlUny on tf^fi Iflral YfO^^ttnn Each elemental downward 'scroU event' can be 
15 achieved dirough the following series of steps: 

(1) left-shift the contents of bi^^O by N(nbits.xmax) bits, where N represents 
the nund)er of lines in each scroll event. (Note that N should be chosen such 
that ymax is an integer multiple of N.) 

(2) copy the first N(nbits.xmax) from bufJ into a temporary 'scroll quanmm' 
20 buffer (not shown). 

(3) write d)e contents of die scrcdl quantum buffer into bitf^O, starting at an 
offset in fti(f, 0 given by (xmax(ymax-N))nbits- 

(3a) write the contents of the scroll quantum buffer to the output comms pon. 

(4) left-shift die contents of bitf^l by N(nbits.xmax) 
75 (5) display die contents of bitf^O 

If it is not necessary to transmit die scrdling viewport to another workstation 
over a LAN or WAN connection, dien step 3a will be omitted. In diis case 
the scroll quantum buffer can also be di^nsed widi &nce the scrcdl quantum 
30 can be copied direcdy from bitf^l to buf^O. 

Steps 1-5 can be repeated to oxitinue scrolling until {ymsx/N) scroll events have been 
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completed. At this point there is no more useful data in buf, 1 and the scroll process 
must pause while a new image is acquired into bi(f^ I . Scrolling can then resume 
as before until a further (ymax/N) scroll events have occurred, etc. If it is desired 
to reduce the slight irregularity which accompanies this type of scrolling, it is a simple 
5 matter to use three buffers instead of two, at die cost of more memory allocation and 
slightly greater complexity In programming. 

In practice, by writing directly to the di?>lay hardware, this process can be 
accomplidicd within 15 msec, caaly meeting the need to refrcdi the display at 30 

10 frames/sec to create the impression of smooth scrolling. To scroll across a full 
viewport dimension in this way takes approximately 12 sec. using compiled Visual 
C++ code, a standard rgb display board and an Intel PI 66 cpu. If two complete 
new lines (a b, c J,-...) are copied to bitfjO in each 'scroll evcm\ tiien the rime 
taken to scan one complete frame height is halved without noticeable loss of 

15 smoothness in scrolling. Further incrcnaenting the 'line quantum' gives a slight but 
noticeable jeridness to the scrolling, which may be acceptable as the price to pay for 
faster scrolling. For a colour image, six image buffers would be required to 
accommodate the colour inoagc planes received from the camera, tiic display buffer 
biifj holding the composite image to be displayed on die monitor at any instant. 

20 

frfimlling ffll mmflffi Wftriffflarion Examination of the nvc steps which make up 
a single local scroll event shows that a similar sequence can be carried out on a 
remote host, but here instead of holding an entire image, bi^J is replaced by die 
input buffer provided by the communications software layer which hdds the 'scroll 
25 quanta' of size N(xmax.nbits) bits. Starting from the same point as before, with bodi 
c(xx4)uters displaying the same initial image from their respective bttfj) memories, 
die sequence on the remote host for a parallel downward scroll event then becomes 

(1) left-shift die contents of tbt remote bitfjO by N(nblis.xmax) bits, where N 

represents the number of lines in each scroll event. 
30 (2) read a 'scroll quantum* of N(nbits.xmax) bits from die comms iqnit data 

stream into the remoce bitfj>, starting at an offset in the remote bitfj> given 

by (xmax(ymax-N)nbits. 
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(3) display the contents of bitfj) . 
In this way the communications link is used to transmit a stream of successive 'scroll 
quanta*. While not essential to the concept of viewport scrolling presented here, this 
method has two advantages compared wiA the alternative strategy of transmitting a 
S succession of entire tnuiges to the remote host: 

a) following im*tiation of the scrolling process via the local host's gui, scrolling 
begins earlier on the remote host. This is because there is no need to wait for 
an entire image to be transmitted before beginning scrolling on the remote 
host. 

10 b) because the image is transnUtted in small ^chunks' rather than as one long 

bitstream, buffer management and error handling are made easier on the 
remote host. 

Movement of the specimen under the objective lens 1 1 in all three orthogonal axes is 
IS brought about through activation of me or more of three stepper-motors provided 
with the motwised stage. The diree stq^r-mocors are driven directly from the 
proprietary stage controller hardware which in turn receives appropriate control 
sgnals in die form ctf ASCII strings via an RS232 pon of the local computer 14. 

20 The microscope stage 10 is controlled by the consultant using a mouse to activate one 
of the four arrow symbols which the gui provides ('scroll fcvward, scroll back, scroll 
left, scroll right*)* The key feature of this type of control is diat the microscope does 
not move smoodily at constant speed: on the contrary, the stage is made to move in 
stq)s corresponding to die absolute height (y^) or width (Xq) of the region in the 

25 viewpcHt frame as appropriate to the selected direction of trawl. For example, 
suppose that die camera/fnunegraM»er In use provides an image (viewport) of height 
768 pixels, and tiiat calibration of die system witis stage mtcrcnneter while usng a 
specific objective lens shows that Imm in the focal plane corresponds io 218 pixels. 
It follows that y^ = 3.S2mm, and this will be progranmied into die startup script 

30 during rtie initial calibration. If now the mouse is used to select die *scroil forward* 
arrow, die stage will be made to move forward in steps of 3.S2mm. Immediately that 
each step is completed (approx 1 sec), an image is grabbed and compressed to RAM 
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(0.5 sec). Oitce an initial stadc image has been transferred to the remote display 
buffer, all that is required is that this buffer is treated in the same way as b^J> in 
Fig. 2, ie it must be updated by incorporation of the 'scroll quanta' which it gets from 
the local ccxnputen 

5 

When one of the 'arrow teys* is acdvaied, then the stage immediately movei by one 
vicwpon height (or width) in Ote af^rcpriatc direction and an image is grabbed to 
bitfj. No further movement of the stage occurs until the viewp<Ht is scrolled 
ctHnplettly out of bttfj), whereupon the stage is made to jump ahead again. 

10 

When scrolling in the x-direcdoo, it is no longer possible to simply shift die endre 
cratents of die store to the right or left when iiK:temennng the stored data. Instead, 
die data representing an incremental strip in die x-dircction must be retrieved from 
each of the yroax lines in the appropriate store and either dissrded or combined with 
15 die data in the conespondlng yniax lines in die odier store to fcmn an up-dated image. 

Odier Implications of the described system will exist wherever readout of data ftom 
a vidcocamera is slow compared to die movement of a fixed scene across die field of 
view. In die present example, die image of die specimen is moved across die CCD 

20 camera chip in a 'saltatory' or 'jumping' modon so that a smoodt, continuous motion 
can be accurately displayed on the VDU. The same principle might be used to move 
die camera itself in this way across a static object to produce a similar effect For 
example, if die camera were made to perform a series of rotations through a fixed 
angular dinance at>out a static axis, pausing on the way to lalce a series of ^c 

25 images of a panoramic scene, then diese images could be processed in die way 
describedabovetoreconstruct die visualdlectofasmoodi panning morion across die 
scene ai almost any desired speed. A similar process could be used to malce smooch 
images in aerial photogiaphy, virtual reality constructions etc.» provided that die 
d)ject in view is not changing appreciably during die period in which die images are 

30 acquired. 
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L Method of displaying a sequence of images from a digital videocamcra 
on a video display unit, the method being 

characterized by the steps of advancing the field of view of the 
camera in discrete steps across the object being viewed to capture a sequence 
of contiguous images, temporarily storing a digital representation of a first of 
the images in a first buffer store and a digital representation of at least part of 
the next succeeding image in a second buffer store, controlling the data in the 
two buffer stores such that the data follows the leading edge of the advancing 
viewport and data in the first store representing a trailing incremental strip of 
the first stored image is progressively discarded while data in the second 
store representing a trailing incremental strip of the next contiguous image is 
transferred to a location in the first store occupied by the data representing a 
leading incremental strip of the first stored image, and reading out the 
contents of the first buffer store to the display unit. 

2. Method according to claim 1, 

characterized in that where the second buffer holds an entire 
image, an intermediate 'scroll' buffer containing the 'scroll quanta' is inserted 
between the first and the second buffers. 

3. Method according to claim 1, 

characterized in that the second buffer contains an entire image, 
that the incrementation is continued until the entire contents of the first 
buffer store representing the original image stored therein have been 
discarded, and that a new image is then grabbed into the second buffer store 
and the process is repeated. 

4. Method according to claim I, 

characterized in that the new image is stored in a third image store 
and that the contents of the second buffer store are then replenished 
continuously with increments from the third buffer store, 

5. Method according to any of the preceding claims, in which the image 
is a colour RGB image, 

characterized in that the first and second buffers are provided for 
each of the three colour components, and that the readout is to a single 
common display buffer. 
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6* Use of a method according to any of the preceding claims in a system 
for transferring pathological information, such as scanned images, &om a 
local to a remote location. 
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